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Technical  Accomplishments 

Under  this  grant  research  is  conducted  at  both  Adaptive  Systems  Inc.  (ASI) 
and  the  Oregon  Graduate  Institute  (OGI).  ASI  is  developing  hardware  im¬ 
plementations  of  models  derived  from  studies  of  olfactory  cortex.  ,For  these 
implementations  the  models  are  being  mapped  onto  the  general- purpose 
neurocomputer  developed  at  ASI.  Research  at  ASI  is  also  directed  towards 
direct  silicon  implementation  of  biologically  faithful  cortex  models. 

Research  at  OGI  is  concerned  with  the  computational  capability  and  the¬ 
oretical  aspects  of  these  models,  as  well  as  with  modifications  that  enhance 
functionality.  We  are  applying  cortex-inspired  models  to  speech  recognition 
problems  as  well  as  pursuing  issues  of  model  convergence  and  computational 
efficiency.  , 
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Implementation  on  High-level  Simulator  (OGI)  The  abstract  cortex 
model  [1]  was  implemented  in  ASI’s  high-level  simulator.  This  was 
the  first  step  in  mapping  the  algorithm  to  ASI’s  neuro-computer.  This 
implementation  provides  a  template  for  the  direct  micro-code  imple¬ 
mentation  on  the  chip.  In  addition,  this  simulator  provided  us  with 
a  vehicle  for  preliminary  studies  of  the  clustering  and  classification 
ability  of  the  model. 

Implementation  in  Micro-Code  (ASI)  We  have  completed  a  direct  micro¬ 
code  implementation  of  the  model  for  the  ASI  neuro-computer.  The 
code  includes  embellishments  to  the  algorithm  developed  by  the  re¬ 
search  group  at  OGI.  Details  of  the  implementation  appear  in  [2], 
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Computational  Capability  (OGI)  In  [3]  we  have  shown  that  the  ab¬ 
stract  model  described  in  [1]  and  implemented  on  the  ASI  neurocom¬ 
puter  is  a  neural  implementation  of  multistage  vector  quantization 
[4,  5].  This  architecture  has  a  significant  storage  advantage  relative 
to  conventional  neural  clustering  and  vector  quantization  (VQ)  algo¬ 
rithms. 

Standard  neural  VQ  algorithms  require  one  neuron  for  each  cluster 
point.  That  is,  each  neuron  responds  to  a  specific  region  of  the  input 
space.  This  defines  the  receptive  field  of  the  neuron. 

The  multi-stage  architecture  defines  cluster  points  by  combinations 
of  neural  receptive  fields.  For  example,  a  cortex  model  with  three 
hierarchical  levels  of  four  neurons  each  generates  64  distinct  cluster 
points  using  only  12  neurons.  A  conventional  competitive  learning 
algorithm  requires  64  neurons  to  define  the  same  number  of  cluster 
points.  Our  experiments  confirm  that  the  cortex  model  when  used 
for  encoding  and  classifying  phonemes  provides  lower  error  per  neuron 
than  either  conventional  or  tree-structured  algorithms  [3], 

Naturally  one  does  not  get  something  for  nothing,  and  there  is  a  trade¬ 
off  inherent  in  the  multi-stage  structure.  Since  each  neuron  in  the 
model  participates  in  defining  several  clusters,  the  latter  cannot  be 
independently  oriented.  However,  the  disadvantage  of  this  constraint 
appears  to  be  overwhelmed  by  the  advantage  gained  by  the  combina¬ 
torial  efficiency. 

Noise  Immunity  (OGI/ ASI)  Noise  immunity  is  an  issue  for  both  bio¬ 
logical  and  limited-precision  hardware  implementations  of  neural  algo¬ 
rithms.  While  many  simulations  are  carried  out  using  double-precision 
floating  point  operations,  biological  systems  cannot  approach  this  level 
of  precision.  Inexpensive  neural  computers  must  forgo  the  luxury  of 
floating  point  processors. 

We  have  conducted  preliminary  studies  using  an  algorithm  that  rescales 
pattern  vectors  to  mock-up  the  approximate  constancy  of  olfactory 
bulb  excitation.  We  find  some  improvement  in  noise  immunity,  with 
further  studies  suggested.  The  current  hardware  implementation  in¬ 
cludes  an  option  to  excercise  this  embellishement. 

Model  Dynamics  (OGI)  We  began  studies  of  the  dynamics  of  self-organizing 
networks  under  a  previous  ONR  grant  and  completed  that  work  under 
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the  present  grant.  (Cliff  Lau  served  as  science  officer  for  the  previous 
grant,  No.  N00014-88-K-0329.)  We  introduced  tools  from  bifurca¬ 
tion  theory  to  treat  the  dynamics  of  learning  in  networks  with  both 
Hebbian  and  anti-Hebbian  synapses  and  recurrent  lateral  connections. 
Our  studies  identify  the  bifurcation  types  and  thus  provide  analytic  de¬ 
scriptions  of  the  location  of  equilibria  and  limit  cycles  in  the  learning 
behavior. 

Our  paper  describing  this  work  was  among  fewer  than  6  %  of  the 
submitted  papers  chosen  for  oral  presentation  at  the  1990  Neural  In¬ 
formation  Processing  Systems  conference  at  Denver,  Colorado  [6].  The 
work  was  recently  published  in  Network  :  Computation  in  Neural  Sys¬ 
tems  [7]  and  is  the  subject  of  invited  talks  at  the  1990  International 
Society  for  the  Systems  Sciences  and  the  1991  SPIE  Conference  on 
Adaptive  Signal  Processing. 

Publications 

Refereed  Papers  —  4 

1.  Leen,  T.K.:  Dynamics  of  learning  in  linear  feature-discovery  networks, 
Network:  Computation  in  Neural  Systems  2,  85,  1991. 

2.  Leen,  T.K.:  Weight-Space  Dynamics  of  Recurrent  Hebbian  Networks, 
to  appear  in  Advances  in  Neural  Information  Processing  Systems  3, 
Morgan  Kauffman,  1991. 

3.  Leen,  T.K.,  Webb,  Max,  Rehfuss,  S.:  Encoding  and  Classification  in 
a  Model  of  Olfactory  Cortex,  to  appear  in  International  Joint  Confer¬ 
ence  on  Neural  Networks,  1991. 

4.  Means,  E.  and  Hammerstrom,  D.:  Piriform  model  execution  on  a  neu¬ 
rocomputer,  to  appear  in  International  Joint  Conference  on  Neural 
Networks,  1991. 

Book  Chapters  -  2 

1.  Hammerstrom,  D.,  Leen,  T.K.,  and  Means,  E.:  Dynamics  and  Im¬ 
plementation  of  Self-Organizing  Networks,  in  Advanced  Neural  Com¬ 
puters,  R.  Eckmiller  (Ed.),  Elsevier  Science  Publishers  B.V.  (North- 
Holland),  March,  1990. 


2.  Hammerstrom,  D.  and  Means,  E.:  A  proposed  architecture  for  a 
second-generation  neurocomputer,  in  Olfaction  as  a  Model  Systems 
for  Computational  Neuroscience ,  Joel  Davis  and  Howard  Eichenbaum 
(Eds.),  MIT  Press,  1991. 

Technical  Reports  and  Non- Refereed  Papers  -  6 

1.  Leen,  T.K.:  Weight  Dynamics  of  Recurrent  Hebbian  Networks,  Pro¬ 
ceedings  of  the  34th  Annual  conference  of  the  International  Society  for 
the  Systems  Sciences,  July,  1990. 

2.  Leen,  T.K.:  Hebbian  Learning:  Algorithms  and  Applications,  Proceed¬ 
ings  of  the  34th  Annual  conference  of  the  International  Society  for  the 
Systems  Sciences,  July,  1990. 

3.  Leen,  T.K.:  Dynamics  of  Learning  in  Recurrent  Hebbian  Networks, 
Oregon  Graduate  Institute  Technical  Report  No.  CSE  90-013 ,  August, 
1990. 

4.  Leen,  T.K.,  Cole,  R.,  Hammerstrom,  D.,  Inouye,  J.  :  Speech  Recogni¬ 
tion  with  a  Cortex  Model  :  Preliminary  Results  and  Outlook,  Oregon 
Graduate  Institute  Technical  Report  No.  CSE  90-022,  June,  1990. 

5.  Leen,  T.K.,  Webb,  Max,  Rehfus,  S.:  Encoding  and  classification  in 
a  model  of  olfactory  cortex,  Oregon  Graduate  Institute  Tech.  Rep. 
CS/E  91-002,  Jan.  1991. 

6.  Leen,T.K.:  Neural  Network  Data  Encoding  and  PCA,  Neural  Network- 
Review,  1991  (in  press). 

Refereed  Papers  Submitted,  Not  Yet  Published  -  1 

1.  Leen,  T.K.,  Webb,  M.,  Rehfuss,  S.:  “Hierarchical  Competitive  Learn¬ 
ing  and  Olfactory  Cortex”,  submitted  to  Neural  Information  Process¬ 
ing  Systems,  1991. 

Invited  Talks  -  2 

1.  Todd  K.  Leen,  “Local  Learning  in  Hebbian  Networks,  R.S.  Dow  Neu¬ 
rological  Sciences”,  Institute,  Feb.  1991. 
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2.  Todd  K.  Leen,  “Bifurcations  in  Learning”,  SPIE  Conference  on  Adap¬ 
tive  Signal  Processing,  to  appear  July,  1991. 

Patents  -  0 
Support 

•  Graduate  Students  Supported  Greater  than  25%  -  2. 

•  Post-doc  support  -  0. 

•  Female,  minority  and  Asian  graduate  students  and  post-docs  -  0. 
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